Docket No.: 0033-0892P 
(PATENT) 

IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

In re Patent Application of: 
Minehiro KONYA et al. 

Application No.: 10/611,871 

Filed: July 3, 2003 

For: MOBILE EQUIPMENT WITH THREE 
DIMENSIONAL DISPLAY FUNCTION 

REPLY BRIEF 

MS Reply Brief - Patents 
Commissioner for Patents 
P.O. Box 1450 
Alexandria, VA 22313-1450 

Sir: 

INTRODUCTORY COMMENTS 

As required under § 41.41(a), this Reply Brief is filed within two months of the 
Examiner's Answer filed in this application on October 2, 2007. 



Confirmation No.: 4164 
Art Unit: 2628 
Examiner: D. F. Hajnik 



Birch, Stewart, Kolasch & Birch, LLP 



Application No. 10/611 ,87 1 

Reply Brief dated November 30, 2007 

Examiner's Answer dated October 2, 2007 



Docket No.: 0033-0892P 



Table of Contents 

This brief contains items under the following headings as required by 37 CF.R. § 41.37 
andM.P.E.P. § 1206: 

I. Status of Claims 3 

II. Grounds of Rejection to be Reviewed on Appeal 4 

III. Arguments 5 



2 



TCB/RWD/tdo 



Application No. 10/611,871 Docket No.: 003 3-0892P 

Reply Brief dated November 30, 2007 
Examiner's Answer dated October 2, 2007 

STATUS OF CLAIMS 

A. Total Number of Claims in Application 
There are 27 claims pending in this application. 

B. Current Status of Claims 

1. Claims canceled: 1-6, 14, 31 

2. Claims withdrawn from consideration but not canceled: NONE 

3. Claims pending: 7-13, 15-30, 32-35 

4. Claims allowed: NONE 

5. Claims rejected: 7-13, 15-30, 32-35 

C. Claims On Appeal 

The claims on Appeal are claims 7-13, 15-30, 32-35 
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GROUNDS OF REJECTION TO BE REVIEWED ON APPEAL 

The following rejections made in the Final Office Action mailed December 27, 2006 (Final 
Office Action) are to be reviewed on Appeal. 

1. Rejection of claims 16, 17, 21, 24-26, and 29 under 35 U.S.C. § 102(a) as being 
anticipated by JP 2002-077944 (Shuji). 

2. Rejection of claims 7-13, 22, 27, and 28 under 35 U.S.C. § 103(a) as being unpatentable 
over Shuji in view of U.S. Publication 2002/0054032 (Aoki). 

3. Rejection of claims 15 and 30 under 35 U.S.C. § 103(a) as being unpatentable over Shuji 
in view of U.S. Patent 6,995,762 (Pavlidis). 

4. Rejection of claim 18-20 under 35 U.S.C. § 103(a) as being unpatentable over Shuji in 
view of U.S. Patent 6,940,646 (Taniguchi). 

5. Rejection of claims 32 and 35 under 35 U.S.C. § 103(a) as being unpatentable over Shuji 
in view of U.S. Patent 5,818,463 (Tao). 

6. Rejection of claims 23, 33, and 34 under 35 U.S.C. § 103(a) as being unpatentable over 
Shuji in view of Aoki, and further in view of Tao. 
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ARGUMENTS 

In the following remarks, Appellants refer to the original Appeal Brief filed April 26, 
2007 (hereinafter "Appeal Brief) and the subsequent Examiner's Answer dated October 2, 2007 
(hereinafter "Examiner's Answer"). 

1. ARGUMENTS FOR REJECTION OF CLAIMS 16, 17, 21, 24-26, AND 29 UNDER 35 
U.S.C. § 102(a) BASED ON SHUJL 

Claim 17 

In the body of the rejection of claim 17, the Examiner has revised the interpretation of 
paragraph 0038 of Shuji, but otherwise substantially repeats the rejection of claim 17. The 
Examiner presents some further arguments for claim 17 on page 20 of the Examiner's Answer. 
In a section "Response to Argument," the Examiner's Answer addresses arguments made in the 
Appeal Brief in paragraph 4 at pages 16-17. Appellants respectfully maintain their position 
regarding claim 17, as well as respective dependent claims. 

Claim 17 recites features of 

"a pickup device picking up a two dimensional image data of a subject," 
"a three dimensional image creation portion" obtaining display data, and 
"a first data process means for generating three dimensional data derived from the two 
dimensional data, and 

a second data process means for converting the three dimensional data into the image 
data for the right eye and the image data for the left eye." 

Regarding the claimed "first processing means," in the section "Response to Argument" 
the Examiner states that: 

"In this reference, the photographic subject is two-dimensional and then is 
converted into a three-dimensional image. The photograph subject becomes three- 
dimensional by adding the parallax information to the image." [Emphasis added] 

Appellants acknowledge the Examiner's interpretation of the Shuji reference. However, 
according to the present specification, "first data process means for generating three dimensional 
data derived from the two dimensional data" corresponds to processor 105 performing the 
function of step S405 in Fig. 4. In particular, according to the specification the processor uses a 
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standard geometry model to generate a three dimensional model corresponding to a texture 
image of the captured two dimensional image (see in particular, original specification at page 8, 
lines 7-9, and at page 9, lines 16-21). 

Furthermore, according to the present specification, "second data process means for 
converting the three dimensional data into the image data for the right eye and the image data for 
the left eye" corresponds to processor 105 performing the functions of steps S406 and S407 in 
Fig. 4. In particular, according to the specification the processor imparts to the three dimensional 
data obtained at step S405 the parallax information (step S406) to create a three dimensional 
image including an image for the right eye and an image for the left eye (S407). (Specification at 
page 10, lines 8-25). 

Thus, according to the disclosure in the present specification, the claimed "second 
process means," and not the "first process means," uses parallax information in providing an 
image for the right eye and an image for the left eye. Furthermore, the claimed "second process 
means" has a function of converting "three dimensional data." 

It is Appellants understanding that Shuji discloses a "three-dimensional scenography 
generation means" that gives different parallax to each part of a photographic subject for 
generation of a three-dimensional scenography of a photographic subject from the captured two- 
dimensional image of the photographic subject. 

In other words, it is Appellants' position that Shuji fails to teach a "first process means" 
and a "second process means" having the respective functions recited in claim 17. Appellants 
submit that because Shuji teaches that two dimensional data becomes three dimensional by 
adding parallax information, Shuji does not disclose the claimed "first process means" 
generating three dimensional data derived from two dimensional data of a picked up image, and 
the claimed "second process means." 

Appellants position regarding the claimed "first data process means" is further amplified 
by the Examiner's arguments regarding the claimed "second data process means." Claim 17 
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requires "second process means for converting the three dimensional data into image data for the 
right eye and the image data for the left eye." In the section "Response to Argument," the 
Examiner states "Shuji teaches of applying an amount of shifting between the left and the right 
eye viewpoints in order to achieve stereoscopic or binocular-like visual, (paragraph [0038] of 
English translation)." Appellants submit that the "amount of shifting" referred to in the 
Examiner's argument for the "second process means" is determined as part of the generation of 
the three-dimensional scenography performed by the three-dimensional scenography generation 
means, which allegedly pertains to the "first process means." 

Furthermore, paragraph 0038 of Shuji discloses that the amount of shift of a pixel in the 
two dimensional image, for the right eye and for the left eye, is according to the distance to a 
photographic subject. In other words, Shuji discloses determination of shift amount for pixels in 
the two-dimensional image . Thus, Appellants submit that Shuji fails to teach the claimed 
"second data processing means for converting the three dimensional data [from the first data 
process means] into the image data for the right eye and the image data for the left eye." 

Thus, as in the Appeal Brief, Appellants submit that Shuji fails to disclose at least the 
claimed "first data process means," as well as the claimed "second data process means." 

The arguments above for claim 17, apply as well to claim 29. Accordingly, Appellants 
request that the rejections of claims 17-22 and 24-29 should be withdrawn. 

Claim 16 

Similar to the case of claim 17, the body of the subject rejection concerning claim 16 
includes a revision to the interpretation of paragraphs 0038 and 0003 of Shuji. The Examiner's 
Answer also presents further arguments for claim 16 in the section "Response to Argument," at 
pages 18 and 19, which address arguments in the Appeal Brief at pages 14 and 15. Appellants 
continue to disagree with the Examiner's interpretation of the teachings of Shuji. 



In the Examiner's Answer at page 19, the Examiner indicates that, 
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"Basic geometry of the setup with these two offset viewpoints (where one is 
shifted relative to another) requires one to have a distance between the eyes and a 
distance to the object being seen by the left and right viewpoint. Since Shuji 
teaches of shifting the view point to given parallax information, the distance to the 
object and the distance between viewpoints (left and right eyes) have to be 
considered and used by the reference in order to achieve correct output results. 
Otherwise, the image may look distorted and no have the correct amount of 
parallax present (i.e. the images may be shifted too little or too much)." 

Appellants maintain that Shuji does not disclose calculation of parallax information based 
on the intensity of light reflected from the subject and on a distance between human eyes. Claim 
16 recites, in the context of a mobile equipment having a pickup device, that "said parallax 
information portion calculates said parallax information based on the intensity of light reflected 
from the subject and on a distance between human eyes." 

As expressed in the Appeal Brief, it is Appellants' position that Shuji teaches that the 
amount of shift for a pixel within a two dimensional image is determined based on distance to 
the photographic subject, (taught, for example, in paragraph 0038 of Shuji). It is Appellants' 
position that Shuji does not disclose calculating parallax information (for example, parallax 
angle) based on the distance between human eyes. Rather, paragraph 0003 of Shuji pertains to a 
prior art stereoscopic camera in which the two cameras take photographs that are photoed with 
spacing equivalent to the parallax of human being's eyes. Paragraph 0003 does not disclose a 
function of calculating parallax information based on the distance between human eyes. 

Appellants submit that Shuji 's imaging device uses an alternative approach based on 
measurements of relative distances. An embodiment of the present invention is for example a 
mobile equipment having a pickup device, and starts with an assumption that there is no or 
limited supplemental equipment such as distance measurement devices. Subsequently, provided 
only a two dimensional image of a subject, the present invention renders stereoscopic images 
using a three dimensional model, calculates approximate parallax information based on estimated 
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parameter values, and converts three dimensional data determined based on the model into an 
image for each eye. 

Shuji discloses a "distance acquisition means" for acquiring distance information for 
bodies to be photographed and a device for generating 3-dimensional scenography from a two 
dimensional image and the distance information. Paragraph 0037 of Shuji states that, "Based on 
the sent usual image and distance information, the image for the right eyes and the image for the 
left eyes are generated by 3-dimensional scenography generation equipment 141." The operation 
of the 3-dimensional scenography generation equipment is described at paragraphs 001 1 to 0015. 
In an example, Shuji discloses that relative to square body lb at a far distance from a camera, a 
round body la that is closer to the camera will be shifted more (paragraph 0012). In other words, 
Shuji discloses that an amount of shift is determined based on the distance between the camera 
and a respective body. In particular, Shuji' s imaging device generates stereoscopic images 
(images having parallax) based on shift in viewpoint (for determining the direction that an object 
should be shifted) and based on this distance between the camera and the object (for determining 
the amount of pixel shift). 

Thus, Shuji discloses an approach that can be applied to any image pickup device using 
the distance acquisition means. Shuji's approach is to determine a relative amount of shift based 
on differences in distance between subjects to thereby achieving a parallax effect sufficient for 
providing a three dimensional display. 

On the other hand, the present claimed invention assumes as a basis a minimal mobile 
equipment to be used to generate a three dimensional display. Assumptions made in the present 
invention include that the object in focus is the object that is the subject of parallax and uses a 
standard distance between human eyes. Applying calculated parallax information to three 
dimensional image information that is determined based on a three dimensional model, 
stereographic images can be created and displayed as a three dimensional display. 
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Therefore, it is Appellants position that Shuji fails to disclose calculation of parallax 
information based on the distance between human eyes. Accordingly, Appellants request that the 
rejection of claim 16 should be withdrawn. 



2. ARGUMENTS FOR REJECTION OF CLAIMS 7-13, 22, 27, AND 28 UNDER 35 U.S.C. § 
1 03(a) BASED ON SHUJI AND AOKI. 

The Examiner's Answer substantially maintains the rejection of claim 7, but revises the 
statement of reasons for combining Shuji and Aoki, which now states that, 



"It would have been obvious to one of ordinary skill in the art to combine Shuji 
and Aoki. Aoki teaches one advantage of the combination by teaching of "a data 
transmission method able to realize communication in a state where eye contact is 
maintained" (paragraph 0015) where Shuji would benefit from such added 
functionality through entertaining and interesting communication." 

The Examiner's Answer also presents further arguments in the section "Response to 
Argument" at pages 21-23, which addresses arguments in the Appeal Brief at pages 18 and 19. In 
place of the teaching, suggestion, motivation criteria for combining references, the Examiner 
relies on a statement from the recent KSR decision that "Common sense can be used by those 
skilled in the art to demonstrate why some combination would have been obvious where others 
would not." KSR, 127 S.Ct. 1727, 1739, 82 USPQ2d 1385, 1395 (2007). 

Appellants believe that the KSR decision does not hold that "common sense" means that 
multiple references can be combined without any basis in fact in order to make a prima facie 
case of obviousness. Instead, it is Appellants' understanding that, 



To establish a prima facie case of obviousness, the prior art reference (or 
references when combined) must teach or suggest all the claim limitations. In re 
Vaeck, 947 F.2d 488, 20 USPQ2d 1438 (Fed. Cir. 1991). Additionally, there must 
be a reason why one of ordinary skill in the art would modify the reference or 
combine reference teachings to obtain the invention. A patent composed of 
several elements is not proved obvious merely by demonstrating that each of its 
elements was, independently, known in the prior art. KSR Int'l Co. v Teleflex Inc., 
82 USPQ2d 1385 (U.S. 2007). There must be a reason that would have prompted 
a person of ordinary skill in the relevant field to combine the elements in the way 
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the claimed new invention does. Id. The Supreme Court of the United States has 
recently held that the "teaching, suggestion, motivation test" is a valid test for 
obviousness, albeit one which cannot be too rigidly applied. Id. Rejections on 
obviousness grounds cannot be sustained by mere conclusory statements; instead, 
there must be some articulated reasoning with some rational underpinning to 
support the legal conclusion of obviousness. Id. 

Appellants submit that the Examiner's reasoning for modifying Shuji with teachings of 
Aoki would not have prompted a person of ordinary skill in the art to combine teachings in Aoki 
and Shuji in the way the claimed invention does. 

Aoki is directed to a data transmission method able to maintain the quality of a 
transmitted image substantially the same as one's own image. (Abstract). Aoki discusses the 
problem of quality in the description of related art, where the problem is stated as that a user who 
is using a terminal can view his own image captured by the camera at a high quality because the 
image is monitored directly, but after compression and transmission to another terminal, the 
quality of the image sent becomes extremely degraded, (para. 0010). Aoki also explains another 
problem that because users at both terminals appear to look down when a camera is located 
above the display, it becomes impossible to maintain eye contact during a conversation, (para. 
0012). Aoki's disclosed solutions propose to cut out only the face image for compression and 
transmission. 

Aoki's disclosed solution to the problem of looking down such that eye contact cannot be 
maintained is to locate the camera lens at a center region in the terminal (see Fig. 3; paras. 0080, 
0153). 

Appellants submit that although it may be desirable to locate a camera at a center region 
of a terminal, modifying Shuji to locate a camera lens at a center region would not have 
prompted one of ordinary skill in the art to combine Aoki's teaching of cutting out a face image 
with the imaging device of Shuji in a manner of the claimed invention. Furthermore, Aoki 
teaches various approaches to cutting out the face in order to reduce the amount of information 
being compressed and transmitted, and by adjusting a face image to fill a frame, high resolution 
background image can be eliminated (e.g., see paras. 0149 - 0151). Aoki refers to the reduction 
in the amount of information to be transferred and adjustment to fill a frame as being an example 
of "high quality" image. 
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Subsequently, Appellants submit that the applying the teachings of Aoki to the invention 
of Shuji would merely involve a specific image to be stereoscopically displayed. In other words, 
the combination of Aoki and Shuji would at most result in obtaining distance information for all 
pixels in a face image in order to display a stereoscopic image of the face image. 

To the contrary, claim 7 requires that parallax information be applied to a cut out of a 
face image in order to create a three dimensional image of the original picked up image. 
Subsequently, Appellants submit that Shuji and Aoki, either alone or in combination, fail to 
teach or suggest the claimed "a three dimensional image creation portion creating a three 
dimensional image by applying said parallax information to said image," and "wherein said three 
dimensional image creation portion cuts a human face out of said image picked up, to obtain a 
face image and provides said face image with said parallax information." 

Furthermore, as indicated in the Appeal Brief, Appellants submit that Aoki fails to make 
up for the deficiency in Shuji of failing to teach a parallax information portion determining 
parallax information of a subject based on a distance between human eyes. 

The arguments above apply as well to steps recited in claim 13, as well as claim 10 
(subject). Accordingly, for the above reasons, Appellants request that the rejection of claims 7-13 
should be withdrawn. 

3. ARGUMENTS FOR REJECTION OF CLAIMS 15 AND 30 UNDER 35 USC 103(a) 
BASED ON SHUJI AND PAVLIDIS. 

The rejection of claim 15 present in the Examiner's Answer has revised the argument 
with respect to the teachings of Pavlidis. The rejection now refers to disclosure of methods of 
estimation of a scale factor, i.e., the scale between the image and the actual object (col. 16, lines 
1-2, which states, "Accordingly, means to convert pixel distances to absolute unites (e.g., inches) 
are provided." Beginning at column 16, line 3, Pavlidis states, Alternatively, the scale of the 
object can be estimated by employing a parallax of a single laser beam." 

Claim 15 is directed to embodiments of the present invention that cover a mobile 
equipment comprising a pickup device, parallax information portion, three dimensional image 
creation portion creating a three dimensional image by applying parallax information to a picked 
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up image, and a display unit. Claim 15 further recites, "wherein said parallax information portion 
calculates said parallax information based on differences in the brightness between parts of the 
image. 

The rejection appears to refer to the two alternative methods as though they are the same 
method. In particular, on page 14 of the Examiner's Answer, the rejection alleges that the second 
method at column 16, lines 15-17, teaches that a bright spot is used to determine distance 
because the bright spot is created through using the two laser beams as described above 
(referring back to the section describing the first method). 

Appellants submit that the rejection demonstrates a misinterpretation of the Pavlidis 
reference, and a misinterpretation of the claimed invention. 

Claim 15 requires calculation of parallax information based on differences in the 
brightness between parts of the image, i.e., not parts of the subject. An advantage of the 
invention of claim 15 is that parallax information can be calculated based solely on information 
contained in the picked up image. 

To the contrary, the "two bright spots" disclosed in Pavlidis are bright spots on an object 
created by the two laser beams (col. 15, lines 57-59). The second alternative that uses a single 
laser beam, produces a single laser spot. In each case, a corresponding image of the laser spots 
can be located (col. 15, line 63, to col. 16, line 1; and col. 16, lines 15-17). However, in neither 
case is a distance calculated based on differences in brightness between parts of the image. 

Thus, Appellants maintain the position that Shuji in view of Pavlidis fails to teach or 
suggest at least the claimed "wherein said parallax information portion calculates said parallax 
information based on differences in the brightness between parts of the image," so that the 
rejection fails to establish prima facie obviousness for claim 15. As noted in the Appeal Brief, 
these arguments apply as well to claim 30. Accordingly, Appellants request that the rejection of 
claims 15 and 30 should be withdrawn. 
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4. ARGUMENTS FOR REJECTION OF CLAIMS 18-20 UNDER 35 U.S.C. § 103(a) BASED 
ON SHUJI AND TANIGUCHI. 

Appellants maintain the position that Taniguchi fails to make up for the above stated 
deficiencies in Shuji with regard to claim 17. 

5. ARGUMENTS FOR REJECTION OF CLAIMS 32 AND 35 UNDER 35 U.S.C. § 103(a) 
BASED ON SHUJI AND TAO. 

See next section. 

6. ARGUMENTS FOR REJECTION OF CLAIMS 23, 33, AND 34 UNDER 35 U.S.C. § 103(a) 
BASED ON SHUJI, AOKI, AND TAO. 

In the Appeal Brief, Appellants had argued claims 23 and 32-35 together based on the 
additional feature recited in the respective dependent claims. 

The Examiner presents some further arguments in the section "Response to Argument" at 
pages 25-27 of the Examiner's Answer in response to arguments presented in the Appeal Brief at 
pages 24-25. 

Appellants disagree with the Examiner's interpretation of the Tao reference with respect 
to the claimed feature of 

"a memory for storing a face geometry model, 

wherein said three dimensional image creation portion further includes 

an extraction means for extracting a human face image data out of the two 

dimensional image data, 

wherein said first data process means generates the three dimensional data from 

the human face image data based on the face geometry model" 

(using claim 23 as an example). 

The Examiner's Answer alleges that because Tao teaches a "face model" at column 5, 
lines 18-20, that Tao teaches the claimed "first data processing means generates the three 
dimensional data from the human face image data based on the face geometry model." 
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Appellants maintain the position that discloser of a "face model" cannot be extrapolated to 
generation of three dimensional data from a human face image based on the "face model." In 
other words, the present claimed invention provides a face geometry model in memory, and uses 
the given face geometry model to generate three dimensional data from a two dimensional image 
of a human face. Appellants submit that Tao does not teach or suggest generation of three 
dimensional data from a two dimensional image of a human face based on a given face model, 
and as stated in the Appeal Brief, subsequent conversion of the three dimensional data into an 
image for the right eye and an image for the left eye. Thus, Tao does not make up for the above 
stated deficiencies in Shuji with respect to claim 17. 

In addition, as noted above, a purpose of the face geometry model is to enable creation of 
a realistic three dimensional display from a minimum of a two dimensional image picked up by a 
pickup device, without supplemental equipment such as a distance measurement device. In 
contrast, Shuji uses a distance acquiring means to acquire distances pertaining to each pixel in a 
picked up image. Instead of using a face geometry model to generate three dimensional data 
from a two dimensional image, Shuji teaches determination of a shift amount of a pixel in the 
two dimensional image according to the distance to the photographic subject. 

For at least these reasons, Appellants request that the rejections of claims 23 and 32-35 
should be withdrawn. 
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Conclusion 

In view of the above remarks, it is believed that claims are allowable. 

Should there be any outstanding matters that need to be resolved in the present 
application, the Examiner is respectfully requested to contact Robert Downs, Reg. No. 48,222 at 
the telephone number of the undersigned below, to conduct an interview in an effort to expedite 
prosecution in connection with the present application. 

If necessary, the Commissioner is hereby authorized in this, concurrent, and future 
replies to charge payment or credit any overpayment to Deposit Account No. 02-2448 for any 
additional fees required under 37.C.F.R. §§1.16 or 1.14; particularly, extension of time fees. 
Dated: November 30, 2007 Respectfully submitted, 

B y^^l^ *&y% 22 z 

Terrell C. Birch 
Registration No.: 19,382 

BIRCH, STEWART, KOLASCH & BIRCH, LLP 
81 10 Gatehouse Road 
Suite 100 East 
P.O. Box 747 

Falls Church, Virginia 22040-0747 
(703) 205-8000 
Attorney for Applicant 
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